* Updated sql_database documentation for resource usage

* updated as per comments

* edit snippet to make more runnable

Tried running the example as is and ran into a few issues:
* dlt[parquet] missing dependency
* missing secret (removed since it's not used in this example)
* removed serialized flag (this is kind of an advanced feature and not really needed here)
* added minimal reflection level, otherwise if you run this script twice in a row you get a "column constraint not supported" kind of error

* move imports into snippet

Co-authored-by: Kenny Ning <[email protected]>

#2070)

* adds engine adapter and passes incremental and engine to query adapter

* adds usage for extended query adapter

* supports custom account host for azure blob storage

* does not wake loader in retry in job

* fixes typing and azure url generation

* fixes docs

* allows subqueries in table adapter and accepts write disposition in sql_table

* fixes type error due to signature detection

* adds table adapter to remove nullability

* fixes subquery table adapter test

* Remove unused imports in Snowflake

* Format the modal example

* allow to select schema from pipeline dataset factory

* fix existing test

…abase (#2077)

* Fixes usage of escaped JSONPath in sql_database incremental cursors

* Docs: add note on escaping special characters in cursor column

* Add an example to the info block

…destinations (#2004)

* add PoC for ibis table support on readabledbapidataset

* add PoC for exposing an ibis backend for a destination

* install ibis dependency for tests

* add support for filesystem

* remove print statments

* remove ibis tables from dbapirelation

* clean up interfaces

* move backend creation and skip tests for unsupported backend

* fix dependencies and typing

* mark import not found, can't be linted on 3.8 and 3.9

* add snowflake and bigquery support

* add redshift and maybe fix linter

* fix linter

* remove unneeded dependency

* add in missing pipeline drop

* fix snowflake table access test

* add mssql support

* enable synapse

* add clickhouse support

* enable motherduck

* post rebase lock file update

* enable motherduck

* add missing ibis framework extras

* remove argument of create ibis backend

* extract destination client factories into dataset file

* fix partial loading example

* fix setting of default schema name in destination config

* fix default dataset for staging destination

* post rebase lockfile update

* always set azure transport connection

* updates docs

* bumps to alpha 1.4.1a0

* fixes mypy linting

* start working on data access docs

* updated page

* small changes

* update snippets to new interface

* fix snippets

* update data access docs

* run grammar checker

* fix links

* Apply suggestions from code review

Co-authored-by: Alena Astrakhantseva <[email protected]>

* fix snippets

---------

Co-authored-by: Alena Astrakhantseva <[email protected]>

* Add ollama config test

Signed-off-by: Marcel Coetzee <[email protected]>

* Merge host and port in single var

Signed-off-by: Marcel Coetzee <[email protected]>

* Remove redundant mypy ingore

Signed-off-by: Marcel Coetzee <[email protected]>

* [test](lancedb): add embedding model env var in test_model_providers

Signed-off-by: Marcel Coetzee <[email protected]>

* [fix](test): remove redundant LanceDB Ollama test case

Signed-off-by: Marcel Coetzee <[email protected]>

* Format

Signed-off-by: Marcel Coetzee <[email protected]>

* [docs](lancedb): update embedding model provider and add custom endpoint support

Signed-off-by: Marcel Coetzee <[email protected]>

---------

Signed-off-by: Marcel Coetzee <[email protected]>

* move notebooks page to run pipelines, move streamlit creds to streamlit page

* remove notebooks page from sidebar

* support sets for pyarrow

…output in load stage (#2100)

* fix error in alive bar

* allow to increate totals on counters

* Add dependencies

Signed-off-by: Marcel Coetzee <[email protected]>

* Add shapely dependency

Signed-off-by: Marcel Coetzee <[email protected]>

* Move sample geodata to correct folder

Signed-off-by: Marcel Coetzee <[email protected]>

* Make smaller

Signed-off-by: Marcel Coetzee <[email protected]>

* Enhance PostgresTableBuilder test suite with geometry type handling.

Signed-off-by: Marcel Coetzee <[email protected]>

* Add tests

Signed-off-by: Marcel Coetzee <[email protected]>

* Add geometry columns with default SRID 4326.

Signed-off-by: Marcel Coetzee <[email protected]>

* resource can't serialize shapely objects

Signed-off-by: Marcel Coetzee <[email protected]>

* Expand geom test

Signed-off-by: Marcel Coetzee <[email protected]>

* Comments

Signed-off-by: Marcel Coetzee <[email protected]>

* Update lock file

Signed-off-by: Marcel Coetzee <[email protected]>

* schema

Signed-off-by: Marcel Coetzee <[email protected]>

* [fix](database): remove unused hex validation method

Signed-off-by: Marcel Coetzee <[email protected]>

* Create custom insert job for geom types

Signed-off-by: Marcel Coetzee <[email protected]>

* Remove hanging client parameter

Signed-off-by: Marcel Coetzee <[email protected]>

* Add a TODO comment to address the issue in the splitting logic

Signed-off-by: Marcel Coetzee <[email protected]>

* Remove unnecessary init override

Signed-off-by: Marcel Coetzee <[email protected]>

* Add debugging points

Signed-off-by: Marcel Coetzee <[email protected]>

* [test](database): add tests for geometry parsing in Postgres

Signed-off-by: Marcel Coetzee <[email protected]>

* Correct row parsing in Postgres destination

Signed-off-by: Marcel Coetzee <[email protected]>

* Yield from supermethod

Signed-off-by: Marcel Coetzee <[email protected]>

* Add control flow for geom

Signed-off-by: Marcel Coetzee <[email protected]>

* Add test

* refactor geo parsing

Signed-off-by: Marcel Coetzee <[email protected]>

* [fix](test): correct schema name in PostGIS geometry test

Signed-off-by: Marcel Coetzee <[email protected]>

* Remove stale test

Signed-off-by: Marcel Coetzee <[email protected]>

* Remove geopandas test until resolution

Signed-off-by: Marcel Coetzee <[email protected]>

* Add docs and raise on malformed values

Signed-off-by: Marcel Coetzee <[email protected]>

* Add postgis dependency to ci

Signed-off-by: Marcel Coetzee <[email protected]>

* fix postgis image repo

Signed-off-by: Marcel Coetzee <[email protected]>

* Add postgis to dbt runner

Signed-off-by: Marcel Coetzee <[email protected]>

* Change snippet to py instead of python

Signed-off-by: Marcel Coetzee <[email protected]>

* add postgis

Signed-off-by: Marcel Coetzee <[email protected]>

* Remove unused geodata file

* Remove unnecessary INSERT class

Signed-off-by: Marcel Coetzee <[email protected]>

* Add WKB format handling

Signed-off-by: Marcel Coetzee <[email protected]>

* Packaging

Signed-off-by: Marcel Coetzee <[email protected]>

* Move import to local

Signed-off-by: Marcel Coetzee <[email protected]>

* Comment

Signed-off-by: Marcel Coetzee <[email protected]>

* postgis docs

Signed-off-by: Marcel Coetzee <[email protected]>

* Update lockfile

Signed-off-by: Marcel Coetzee <[email protected]>

* fix(deps): remove shapely dependency from postgis extra

Signed-off-by: Marcel Coetzee <[email protected]>

* format

Signed-off-by: Marcel Coetzee <[email protected]>

* feat(postgres): add support for CSV loading of geometry columns

Signed-off-by: Marcel Coetzee <[email protected]>

* Remove wkb examples in docs

Signed-off-by: Marcel Coetzee <[email protected]>

* format

Signed-off-by: Marcel Coetzee <[email protected]>

---------

Signed-off-by: Marcel Coetzee <[email protected]>

* Incremental table hints and incremental in resource decorator

* Extract incremental settings to a dict in table schema
* Support passing incremental settings to @resource decorator

* Fix type errors

* Reset incremental from_hints when set in resource decorator

* Column hint

* Merge multiple hints

* Test non column match

* adds make_hints test

* Accept TIncrementalconfig in make hints

* bool only incremental hint

* Test jsonpath simple field name

---------

Co-authored-by: Marcin Rudolf <[email protected]>

* removes astunparse and aiohttp

* allows for built-in ast unparse if present

* uses break path for normalization to allow names containing path separators, migrates old schema to enable compat mode with old behavior

* adds removeprefix util

* updates docs

* bumps dlt to version 1.4.1

* linter fixes

* fixes tests

* fixes and tests saving pandas indexes

* fixes sqllite read interface tests

* updates docs